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DEVELOPMENT  OF  NEW  SELECTION 
TESTS  FOR  AIR  TRAFFIC  CONTROLLERS 


This  re|)ort  (lescril)es  oi'ention  of  the  Multiplex 
Controller  Aptitude  Test,  a new  measure  to 
screen  applicants  for  Federal  Aviation  Admin- 
istration Air  Traffic  Controller  imsitions.  Its 
content  includes  items  to  measure  the  traditional 
tv|jes  of  aptitmles  such  as  arithmetic  i'easonin<r 
and  visualization  that  ai'e  included  in  today’s 
Civil  Service  Commission  tests.  In  addition  it 
includes  new  job  sample  items  from  the  con- 
troller activity,  uses  fi^rui-es  to  .show  air  traffic 
on  a simulated  radar  screen,  and  requires  iden- 
tification of  jjotential  conflicts  betAveen  aircraft 
in  that  simulated  traffic.  All  test  questions  are 
presented  in  an  air  traffic  control  setting,  provid- 
inp  a job-related  apiiearance  that  pives  the  test 
hiph  face  validity  not  found  in  today’s  selection 
batterj'.  The  fonnat  for  item  sequencinp  also  is 
a departure  from  another  current  aptitude  bat- 
tery desipn  practice,  that  of  clusterinp  items 
into  homopeneous  subpi-oups.  Instead,  in  these 
tests  the  items  alternate  from  one  type  to  another 
and  spiral  to  increasinp  levels  of  difficulty,  a 
mode  of  presentation  found  only  in  a few  cur- 
rent tests  such  ns  the  Standford-Binet.  A result 
shown  statistically  is  that  non-confliction  items 
included  for  measurement  of  aptitudes  repre- 
sented in  today’s  selection  batterv’  shoAv  unex- 
pectedly hiph  commonality  with  the  new  air 
traffic  items  requirinp  detection  of  imjAendinp 
conflictions.  The  resultinp  measure  shows 
promise  of  consistently  producinp  hipher  validity 
coefficients  than  possible  with  the  current  test 
battery. 

Backpround 

The  current  Civil  Sendee  Commis.sion  aptitude 
te.st  battery  for  screeninp  air  traffic  conti-oller 
applicants  has  lieen  oix*rational  since  1D64.  In 
1970  the  Federal  Aviation  Administration  con- 
tracted with  specialists  in  the  field  of  {wrsonnel 
.selection  to  search  for  ways  to  alleviate  pi-oblems 
beinp  experienced  in  selection  and  retention  of 
air  traffic  controllers.  Thev  identified  other  avail- 


able tests  to  increase  the  pi’edictive  validity  of 
this  battery,  plus  areas  for  the  construction  of 
new  tests.  One  of  these  (Buckley.  Xote  1)  made 
use  of  a techni<|ue  developed  to  evaluate  total 
man-machine  system  i)erformance  in  control  of 
air  traffic  by  measurinp  how  well  a fidly  trained 
specialist  could  control  simulated  traffic  presented 
on  a film  usinp  the  proposed  desipns.  In  an 
adaption  of  this  technique  for  the  testinp  of 
jAersonnel  .skills,  films  of  air  traffic  as  seen  on  a 
radar  scoj>e  are  presented,  and  observera  asked 
to  identify  the  aircraft  they  predict  will  violate 
separation  standards.  This  instniment  added 
sipnificantly  to  a composite  for  predictinp  on- 
the-job  success  (Milne,  Xote  2).  but  the  equip- 
ment and  space  required  for  its  administration 
penerally  are  not  available  in  Civil  .Sendee  Com- 
mission testinp  situations.  Therefore  test  de- 
velopment procedures  wei-e  initiated  to  derive 
measures  of  this  same  skill  in  a format  that  meets 
Civil  Service  needs. 

Test  Development 

The  oripinal  Controller  Decision  Evaluation 
technique  (Buckley,  Xote  1)  required  ohservation 
of  simulated  air  traffic  situations  as  they  unfold 
in  a film  of  a radar  scope.  The  ohservev  pre- 
dicts and  records  potential  conflicts  as  soon  as 
he  detects  them,  in  an  unstructured,  free  resimnst* 
mode.  During  exj>erimental  administration  and 
study  of  three  film  versions  for  development  of 
the  Multiplex  Controller  Aptitude  Test,  it  was 
found  that  identification  of  imtential  conflicts 
Avas  ea.sier.  quicker  and  more  accurate  after  ob- 
serA'ers  adjusted  to  the  scoik*  and  its  targets.  Avhen 
only  a feAv  targets  were  in  near-confliction,  and 
Avhen  their  rate  of  closiire  Avas  sIoav.  Items  Avere 
more  difficult  Avhen  the  obserAera  AA’ere  first  ex- 
l)osed  to  the  scope  and  its  targets,  when  multiple 
targets  AA-ere  in  near  confliction  and  AA-lien  the 
rate  of  closiire  among  targets  Avas  rapid.  False 
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positive  conflict  items  did  not  discriminate  and  eacli  other  and  the  correct  answers  became  more  3 

^nerally,  for  true  conflicts,  the  ^rivater  the  lead  obvious.  In  the  free  response  film  version,  this 

time  available,  the  frreater  an  item's  discriminat-  problem  was  controlled  by  a requirement  for  ^ 

in}r  power.  Journeyman  controllers  identified  entries  from  a coded  clock  onto  the  computer- 

potential  conflicts  sooner  than  developmental  scored  answer  sheet  whenever  potential  conflicts  < 

trainees,  and  identified  almost  all  potentials  that  were  reported.  It  was  found  with  the  paper  and 

became  real  conflicts.  Devel(.,unental  controllers  pencil  vereion  that  presentation  of  a new  item 


fjenerally  missed  a numl)er  of  real  conflicts,  were 
slower  than  journeymen  in  calling  them  out,  and 
frequently  did  not  attend  to  aircraft  altitude 
separations.  However,  the  rantre  in  test  iterform- 
ance  amonp  developmentals  was  fjreat.  and  some 
perfonned  as  well  as  journeymen.  .Some  con- 
fliction  items  in  these  films  were  too  easy  and 
some  had  a negative  relation  to  a developmental- 
journeyman  criterion  dichotomy. 

The  observer  had  much  idle  time  durinfi  free 
response  film  versions  of  the  test,  as  the  traffic 
presented  is  light  and  aircraft  widely  separated. 
In  this  sense,  the  .setting  resembles  a common  air 
traffic  situation,  but  an  extended  amount  of  test- 
ing time  is  required  to  present  only  a small 
number  of  confliction  items.  Better  use  of  ex- 
aminees’ time  was  desired.  An  initial  change 
during  test  development  was  to  structure  the 
items,  using  slides  to  present  questions  on  the 
screen  above  the  film.  Mean  response  times  for 
developmental  and  journeyman  controllers  to 
react  to  each  conflict  in  the  free  response  ver- 
sions were  determined.  Journeyman  mean  times 
were  selected  to  be  determiners  of  aircraft  posi- 
tions when  presenting  confliction  questions  in 
the  stnictured  versions,  for  this  maximized  the 
discriminating  power  of  each  item.  Two-choice 
conflict  items,  asking  whether  a pair  of  aircraft 
would  conflict,  were  presented  for  thirty  seconds 
when  unexpected  changes  occurred  among  tar- 
gets. .such  as  when  new  aircraft  entered  the  pic- 
ture. Four-choice  conflict  items,  asking  which 
pair  would  conflict  if  a confliction  occurs,  wei-e 
presented  for  forty-five  seconds  when  traffic 
changes  were  slow.  A “None  of  these”  response 
was  introduced  to  permit  inclusion  of  non-con- 
flictions  that  might  Iw  predicted  to  lx*  conflic- 
tions.  and  this  was  used  ns  the  forth  alternative 
in  all  four-choice  confliction  items.  Items  were 
a.s.sembled  into  the  described  test  format,  but  they 
didn't  utilize  half  the  available  testing  time.  This 
large  amount  of  idle  time  between  questions  also 
provided  much  opiwrtunity  for  observers  to 


every  45  seconds  would  keep  examinees  so  busy 
that  they  would  have  nf)  opimrtunity  to  make 
such  changes.  More,  test  items  were  needed  to 
make  this  control  procedure  work. 

The  test  films  presented  simulated  traffic  mov- 
ing across  a radar  scope  plus  a table  which  in- 
cludes detail  data  on  each  aircraft,  identifying 
the  target,  its  altitude,  speed,  and  route  on  the 
scope.  The  scope  uses  lines  to  represent  airways 
or  highways  in  the  sky,  with  alphalwtical  iden- 
tifiers of  starting,  ending,  and  intersecting  points 
on  the  airways.  The  top  of  the  scope  is  North, 
and  a mileage  scale  is  provided  at  the  l>ottom. 
Ample  information  is  presented  to  prepare  a 
variety  of  (jiiestions  related  to  a controller  ac- 
tivity. It  appeared  pos.sible  that  most  factore 
used  in  the  current  Civil  Service  Commission 
aptitude  battery  for  controller  selection  could 
be  measured  within  this  simulated  air  traffic 
setting. 

Items  were  written  utilizing  this  available 
information  to  measure  .such  aptitudes  as  direc- 
tion following,  table  reading,  interpretation  of 
data,  spatial  visualization  and  orientation,  esti- 
mation of  distances  and  relative  target  move- 
ments, and  arithmetic.  Some  items  included 
were  of  a very  simple  nature,  and  others  were 
written  in  a multi-factor  format  to  increase  their 
level  of  difficulty.  For  example,  the  directions 
for  this  type  of  test  required  instruction  on  how 
to  read  the  table,  so  initial  table  reading  ques- 
tions were  included  that  were  of  the  very  easy, 
instructional  tyi)e.  .Vwareness  of  distances  across 
the  scope,  reading  the  table  to  determine  the 
speeds  of  aircraft,  and  mathematical  computa- 
tion to  determine  their  rates  of  closure  (all  of 
these  Iwing  related  to  horizontal  separation)  were 
required  to  solve  a complex  problem  such  as 
estimating  the  travel  time  in  minutes  lietween 
two  aircraf'i  at  a given  moment. 

The  relation  lietween  item  types  and  total  test 
homogeneity  was  determined,  and  this  ratio  was 


change  their  anwers  to  earlier  questions  regard- 
ing potential  conflicts  as  the  aircraft  approached 


used  to  determine  the  numlier  of  items  jier  tyjie 
to  include  in  the  test.  A result,  for  example. 
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was  inclusion  of  twice  as  many  tarpet  time-dis- 
tance separation  items  as  compass  headinp  items 
in  the  test.  Tlie  order  of  placement  of  conflict 
items  had  ah'eady  been  established  by  using  the 
mean  response  time  of  joumeyman  controllers 
when  targets  were  at  certain  location.  Xew  apti- 
tude test  items  were  placed  in  the  remaining 
positions,  alternating  from  one  tyiie  to  another 
and  spiraling  to  increasing  levels  of  difficulty 
as  testing  progressed.  Three  alternate  forms  of 
the  test,  MCAT  4,  6,  and  7,  were  prepared. 

Test  Criterion  Characteristics 

Versions  of  the  free  res|>onse  CODE  test,  the 
structured  Multiplex  Controller  Aptitude  Test, 
and  other  selection  measures  were  administered 
to  students  at  the  US  Navy  Air  Traffic  Controller 
Training  School,  Memphis,  during  the  week  of 
August  18-22,  1975.  Grades  of  these  students 
were  obtained  as  they  progressed  through  class- 
room and  laboratory  training  and  passed  or 


failed  the  course.  Distribution  statistics,  inter- 
correlations  and  rotated  factor  loadings  of  the 
various  tests  with  each  other  and  with  school 
grades  were  detennined.  Forty-eight  variables 
were  involved;  five  factors  were  extracted  and 
rotated.  Definitions  of  the  rotated  factors  are 
presented  below. 

Rotated  Factor  1.  Variables  with  highest 
loadings  on  this  factor  include  CTO  Block  I 
Average,  .76,  ITEX  Final  Criterion  P^xam,  .54, 
and  Control  Tower  Training,  Airport,  .53.  This 
factor  might  be  defined  as  Control  Tower  Op- 
erator, the  title  given  by  the  school  to  those 
classroom  grades  with  highest  loadings  or  [jortion 
of  variance  on  this  factor.  Among  potential 
screening  measures,  a new  P'AA  air  traffic  con- 
troller ATC  Occupational  Knowledge  Test  I has 
the  highest  loading,  .40. 

Rotated  Factor  II.  Variables  with  highest 
loadings  on  this  factor  include  Spatial  Test.  .65. 


Table  1 

Intercorrelations  of  Selected  Test  Variables  and  Laboratory  urades 
USN  Air  Traffic  Control  Training  School 
(109  Persons  Tested  August  18-22,  1975) 


Laboratory  Courses 

Code 

MCAT 

OKT-1 

GCT 

Arithmetic 

Reasoning 

Laboratory  Flight  Plans 

VFR 

05 

17 

23 

10 

05 

IFR 

18 

26 

22 

19 

26 

Stopover  Composite 

09 

20 

28 

26 

09 

Performance  Run 

19 

22 

22 

25 

18 

Laboratory  Control  Tower 

Training 

Basic 

16 

20 

28 

19 

21 

Intermediate 

19 

11 

07 

17 

09 

Advanced 

26 

17 

27 

19 

08 

Laboratory  Radar  Training 

Air  Surveillance 

Week-1 

-02 

10 

11 

17 

10 

Week-2 

-08 

04 

21 

09 

12 

Precision  Approach 

Week- 3 

08 

21 

13 

14 

08 

Week-4 

16 

04 

09 

11 

25 

Course  Average 

22 

27 

43 

34 

23 

Free  Response  Code 

43 

32 

12 

14 

MCAT 

42 

20 

23 

ATC  Occupational  Knowledge 

29 

24 

USN  GCT 

31 
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Mechanical  Comprehension,  .63,  and  Arithmetic 
Reasoning,  .53.  This  factor  might  be  defined  as 
Mechanical-Spatial,  .since  the  spatial  and  me- 
chanical comprehension  selection  tests  have 
highest  loadings  on  tliis  factor.  This  factor 
seems  to  account  for  veiy  little  criterion  variance. 

Rotated  Factor  III.  Variables  with  highest 
loadings  on  this  factor  include  Course  Average, 
.82,  Basic  Lab.  Control  Tower  Training,  .65,  an<l 
Stopover  Composite  Flight  Plan  I^ab,  .60.  This 
factor  might  be  defined  as  Ijaboratory  Perform- 
ance, since  laboratory  activities  liave  their  highe.st 
loadings  on  this  factor.  Among  potential  screen- 
ing measures,  ATC  Occupational  Knowledge 
Test  I and  the  Multiplex  Controller  Aptitude 
Test,  with  loadings  of  .22  and  .21  resj)ectively, 
have  the  highest  loadings  on  this  factor. 

Rotated  Factor  IV.  Variables  with  highest 
loadings  on  this  factor  include  Multiplex  Con- 
troller Aptitude  Test,  .58,  Free  Response  Code. 
.49,  ATC  Occupational  Knowledge  Test  I,  .46, 
Xavy  General  Classification  Test,  .40,  ACT  Occu- 
pational Knowledge  Test  II,  .36,  Charts  & Pub- 
lications, Base  Ops.,  .36,  and  Precision  Approach 
Radar  Lab.,  .30.  This  factor  might  lx*  defined 
as  Air  Traffic  Controller  Performance,  since 
selection  tests  involving  the  controller  activity 


and  classroom  and  laboratory  performance  vari- 
ables have  significant  loadings  on  this  factor. 

Rotated  Factor  V.  Variables  with  highest 
loadings  on  this  factor  include  Reading,  .70, 
Radioman,  .58,  and  Precision  Approach  Radar 
Lab.,  .52.  This  factor  might  be  defined  as  Read- 
ing, since  that  variable  has  the  highest  loadings 
on  this  factor. 

Selected  Intercorrelations 

Selected  data  are  presented  in  Table  1 to  high- 
light three  of  the  experimental  selection  tests 
l)eing  studie<l  here,  their  interrelations  with  each 
other  and  with  laboratory  Flight  Plans,  Control 
Tower,  and  Radar  Training,  as  those  grades  pro- 
vide the  greatest  range  in  criterion  scores  for 
differentiating  iierformance  among  FAA  student 
controllers.  ATC  Occupational  Knowledge  Test 
I,  a measure  of  each  person’s  past  controller  ex- 
I)erience,  showed  highest  relation  to  performance 
in  lal)oratorj’  classes.  The  thirty  minute  Multi- 
plex Controller  Aptitude  Test  generally  had  cor- 
relations with  criterion  variables  that  are  as  high 
or  higher  than  those  between  the  Free  Resimnse 
CODE  test  and  those  same  criterion  variables. 
These  results  encouraged  further  development  of 
the  Multiplex  Test. 


Selection  Teet  Variable*  and  School  Grades 
Selected  Intercorrelation  Data 
USAF  Air  Traffic  Control  Training  School 
(461  Person*  Tested  October  20-24,  1975) 


'est  and  Course  Variables 


1.  Final  Score  in  Block-  I 

2.  II 

3.  Ill 

4.  IV 

5.  No.  Counseling  Sessions 

6.  CODE  7-  (Free  Response) 

7.  MCAT  606AS 

8.  606FS 

9.  706FS 

10.  406FS 

11.  Occupational  Knowledge  Test-  I 


*No  Case* 


34  41 

49 


4 

5 

6 

7 

8 

9 

10 

11 

12 

39 

-32 

08 

35 

15 

20 

19 

42 

-01 

31 

-33 

07 

21 

15 

21 

30 

IS 

09 

45 

-36 

28 

33 

16 

17 

27 

32 

39 

-51 

28 

06 

18 

07 

01 

68 

01 

-12 

-25 

02 

-05 

-04 

11 

13 

* 

28 

e 

* 

• 

e 

* 

61 

59 

• 

* 

66 

• 

52 

38 

57 

41 

20 

Media  For  Presenting  Items 

A next  step  in  test  development  Avas  to  con- 
v’ert  the  slides  plus  film  presenting  moving  tar- 
gets to  a “slide  only”  presentation,  in  essence 
capturing  pictni'es  of  the  scope  with  targets  in 
the  same,  position  as  when  each  question  appeared 
on  the  screen.  Pacing  of  slide  presentation  was 
the  same  as  in  the  film  phis  slide  vereion.  so  the 
amount  of  target  movement  from  <|uestion  to 
question  was  unchanged.  This  format  was  easier 
for  test  administi'atoi's  since  they  no  longer  had 
to  cope  with  film  projection  prohlems.  and  the 
stationary  targets  presented  by  slide  seemed 
easier  to  read  tiian  tlie  moving  targets  presented 
by  film.  Persons  taking  the  experimental  ver- 
sion in  this  slide  format  did  grumble  that  the 
test  seemed  to  take  control  over  their  time,  as 
each  item  had  to  lie  completed  quickly  before  it 
left  the  screen  and  a new  item  appeared.  This 
task  characteristic  has  some  commonality  with 
controller  situations  in  the  real  world,  as  traffic 
movement  tends  to  control  their  involvement  and 
pace  of  work.  Another  merit  of  this  version 
would  be  its  case  for  transition  into  a jiaper  and 

Table  3 


pencil  fonnat  that  would  meet  Civil  Sendee 
Coinmis.sion  requirements  for  use  in  their  very 
decenti’alized  field  testing  situations.  F'ilms  plus 
slide  and  “slide  only”  versions  of  the  Multiplex 
Controller  Aptitude  Test  plus  other  selection 
measures  were  administered  to  students  at  the 
USAF  Air  Traffic  Controller  Technical  School. 
Keesler  Air  Force  Base.  Mississippi,  (hiring  the 
week  of  October  20-ii’4.  1975.  Orades  of  these 
students  were  obtained  as  they  progressed 
through  classroom  and  lalioratory  training  and 
passed  or  failed  the  course.  Distribution  statistics 
and  intercorrelations  of  the  various  tests  with 
school  grades  were  determined.  Twenty-one 
variables  were  included,  and  selected  data  are 
presented  in  Table  2.  ATC  Occuiiational  Knowl- 
edge. Test  I,  a measure  of  tlie  individual's  past 
controller  exi)erience.  shows  highest  selection  test 
relation  to  final  score's  in  training.  The  slide 
A^ei-sion  of  Multiplex  generally  shows  higher  cor- 
relations with  the  criteria.  Block  I-IV  Final 
Score,  than  CODE  7 (Free  Response)  and  the 
combination  film-slide  vei’sions  of  MCAT. 


Multiplex  Controller  Aptitude  Test 
Distribution  Statistics  for  Various  Populations 
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.V  next  stop  in  tost  dovolopinont  was  to  print 
tlio  sliilo  vorsions  in  pa|>or  and  iH>noil  format. 
Tlio  slido  vorsion  and  ooinbination  film-slido  vor- 
sion  iiorinittoil  tinio  oontrol  at  tho  individnal  item 
lovel,  allowinjr  thirty  sooonds  for  response  to  two- 
ohoice  items  and  forty-five  seconds  for  response 
to  fonr-clioico  items.  One  paiM*r  and  (tencil  ver- 
sion allows  uninterrupted  work  thron<rhont  test- 
in<r  time,  with  announcements  when  fifteen 
minutes  Itave  elapsed  and  witen  only  five  minutes 
remain  to  work  on  the  test.  A st'jrmented  paper 
and  itencil  version  allows  five  minutes  for  re- 
sponse to  each  cluster  of  items  presented,  with 
a cluster  containin<r  as  many  as  eleven  items  when 
they  are  all  of  the  two-choice  type.  Three 
parallel  forms  of  the  written  test  were  prepared. 
^rCAT  406  including  41  items.  606  containin" 
4.S  items  and  706  containin"  ii.S  items.  Time 
limits  are  25  minutes  plus  directions  for  MCAT 
406  and  606  and  30  minutes  plus  directions  for 
ilCAT  706.  Each  form  now  has  heen  extended 
to  55  items.  IMCAT  407  includes  23  conflict  and 
32  aptitude  items.  MCAT  607  contains  22  conflict 
and  33  aptitude  items,  and  MCAT  707  contains 
23  conflict  and  .32  aptitude  items.  Time  limits 
aiv  35  minutes  for  each  of  tliese  tests. 

Reliability 

The  questions  presented  in  Multiplex  Con- 
troller Aptitude  Test.  Form  4.  6 and  7.  remained 
unchanjred  for  some  length  of  time.  thou"h  the 
media  used  for  presentin"  items  chanfrcd  from 
film  to  slide  to  paper  and  pencil.  This  chnn"e 
in  media  for  presentinjr  items  has  some  effect  on 
an  individual’s  test  performance,  hut  all  forms 
are  qualitatively  similar.  Correlations  of  these 
parallel  forms  provide  a minimum  estimate  of 
the  measure’s  test-retest  reliability  when  admin- 
istered to  members  of  a similar  population. 

These  fonus  have  heen  administered  to  stu- 
dents of  the  nSAF  Air  Traffic  Controllei’  Tech- 
nical School  in  all  jAiases  of  training  and  to  a 
sample  of  forty  non-controller  hisrh  school  and 
collepe  level  FA.\  employees,  "roups  whose 
abilities  approximate  those  of  the  general  popu- 
lation of  applicants  that  take  the  Civil  Service 
Commission  tests.  They  also  have  lieen  admin- 
istered to  entering  FAA  .Vir  Traffic  Controller 
.Vcademy  students  in  the  new  centralized  Air 
Tralfic  Controller  traininfr  propram.  i>ei-sons 
selected  off  the  toj)  of  the  Civil  Service  Register 


for  controller  applicants.  Table  3 presents  dis- 
tribiitioti  statistics  for  the.se  groups  on  the  vari- 
ous forms  of  the  test.  The  range  in  test  scores 
is  highly  l•estricted  for  the  ATC  Academy  group 
when  comparod  to  the  range  for  the  moi-e  general 
population. 

Cori-elations  lietween  forms  were  lowest.  .31  to 
.50.  for  the  ATC’  Academy  students  who  had  a 
gi-eater  i-estriction  in  range  of  their  scoies.  They 
I'ange  from  .60  to  .66  for  the  USAF  ATC  stu- 
dents and  .87  to  .!H)  for  non -controller  FAA 
employees  whose  abilities  more  nearly  approxi- 
mate that  impulation  for  whom  the  test  is  de- 
signed. These  coefficients  indicate  that  the  test 
will  provide  reliable  measui’es  of  jierformance 
under  well  controlled  conditions.  b\it  a combin- 
ing of  items  from  two  forms  would  provide  a 
more  satisfactory  measure  to  use  in  an  opera- 
tional testing  situation. 

Test  Validity 

For  test  validation  purposes  the  criterion  used 
was  “success”  in  air  traffic  control  work.  ."Suc- 
cess as  defined  here  for  ATC  applicants  is 
hierarchical,  including  (1)  satisfactory  comple- 
tion of  the  initial,  formal  training  program, 
(2)  satisfactory  performance  on  the  job.  and 
(S)  progression  or  upward  mobility  within  the 
.VTC  system.  Another  element.  (4)  attrition, 
may  l)e  a measiue  of  non-success.  Those  enrolled 
in  the  initial,  format  training  program  are  a 
highly  select  group,  for  they  scoied  high  enough 
on  the  Civil  Service  Commission  tests  to  lie  hired, 
but  group  memtiership  liecoines  even  more  selec- 
tive as  those  who  fail  to  learn  or  perform  ade- 
(piately  are  separated.  This  selection  [u-ocess 
continues  as  they  progress  up  the  ATC.S  caiwr 
ladder  and  are  evaluated  for  satisfactory  per- 
formance on  the  job  and  advancement  into  the 
more  demanding  and  higher  paid  positions  with- 
in the  .VTC  system.  The  range  in  scores  on 
these  selection  tests  will  lie  greatest  for  the  group 
when  tliey  first  enter  formal  training,  but  liecome 
ever  moiv  restrictive  with  career  progression. 
.Vs  this  restriction  in  range  may  have  a direct 
effect  on  the  size  of  the  validity  coefficient.s.  the 
validities  shotdd  lie  highest  wlien  the  group  is 
in  initial  training  and  become  lower  for  that 
liortion  of  the  group  which  progresses  up  the 
carrer  ladder. 
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Table  4 


BEST  AVAIL', EiP  rm 


Correlations  Between  Selection  Tests  and  Various  Criteria 


Training  Scores 
Terminal  Option 
Enroute  Option 
Both 

b 

Supervisory  Rating 

Flight  Service  Station 
Terminal  VFR 
IFR 

Enroute 

All  Options 

ATC  Progression^ 

Flight  Service  Station 
Terminal  VFR 
IFR 

Enroute 

All  Options 

d 

Aggregate  Criterion 

Flight  Service  Station 
Terminal  VFR 
IFR 

Enroute 

All  Options 


Number 
of  Cases 


301-310 

257-263 

558-573 


195-245 

169-190 

188-188 

181-241 

733-833 


157-159 

191-193 

179-179 

199-200 

727-731 


196-249 

479-518 

499-623 

445-514 

1309-1603 


Selection  Tests 

Arithmetic  MCAT  Directional  Dial 

Reasoninu  Conflicts  Aptitudes  Headinss  Readtne 


Selected 
Batter 


Note.  Decimals  Omitted  '^£<•05  #*£<.01 

Note.  Data  are  from  Mies,  J.  M, , Colmen,  J,  G,  and  Domenech,  O.  , May  1977. 
a Table  XI.  1 
b Table  XII.  2 
c Table  XIII.  2 
d Table  XV.  2 


Success  in  Training.  A battery  of  predictors 
including  Multiplex  Controller  Aptitude  Test 
(i06A  and  TOGAS,  scored  separately  for  Conflicts 
and  Aptitudes,  plus  Directional  Headings,  Dial 
Reading  and  Arithmetic  Reasoning  tests  liecause 
of  the  promise  they  demonstrated  in  an  earlier 
study  (Milne,  Xote  2),  were  administered  to  the 
•lamiary  1076  Class  of  students  entering  the  ATC 
Academy  centralized  training  course.  Persistent 
mechanical  problems  in  operation  of  film,  slide 
and  slide  programmer  equipment  gave  strong 
support  to  the  Civil  Service  Commission’s  stated 
need  for  development  of  these  measures  in  a 
palmer  and  i)encil  foniiat.  Thus  the  March  1976 
class  of  students  were  administered  only  paper 
and  i)encil  vei-sions  MCAT  706A  and  GOfiB.  ])lus 
the  other  selection  measures.  .Satisfactory  com- 
pletion of  the  initial,  formal  training  program 
for  those  enrolled  in  the  Enroute  option  was 
based  on  scores  received  on  four  ATC  lnl>oratory 
problems  plus  scores  on  a Controller  Skills  Test. 
For  those  enrolled  in  the  Terminal  coui'se.  |)er- 
formance  was  based  onlv  on  the  ATC  lal)oratorv 


problems.  These  scoi’es  roquiro  students  to  dem- 
onstrate operational  application  of  academic 
knowledge.  Cori-elations  l)etween  selection  tests 
and  training  scores  ai-e  pi-esented  in  Table  4. 
The  MCAT  Conflict  and  Autitude  scoros  con- 
stantly produced  higher  validities  than  any  other 
test  used.  In  fact,  addition  of  the  other  tests 
increased  the  multiple  con-elation  by  only  .017 
for  Terminal,  .012  for  Enroute.  and  .008  for  Iskh 
options  combined. 

Performance  on  The  Job.  Test  validation  in 
the  oi)erational  .situation,  test  con-elation  with 
s)ii>ervisoi-y  ratings  and  cai-eer  pi-ogi-ession  scoi-es. 
was  accomplished  as  part  of  a larger  ATC  selec  - 
tion research  effort  (Mies,  Xote  3).  A compiv- 
hensive  sampling  design  was  constructed  to 
define  the  ATC  population,  constraints  weiv 
established,  and  stratified  random  sampling 
methods  then  applied  to  select  the  priman- 
samples  of  specialists  from  within  the  total  avail- 
able AT(’  population.  On-the-job  performance 
was  measui-ed  by  confidential  job-task  assess- 
ments pi-epai-ed  by  each  employee’s  suiiervisoi-. 
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Table  5 


Integer  Weights  Assigned  to  Tests 
By  ATC  Option  and  All  Options  Combined 


ATC  Options 

MCAT 

Conflicts 

MCAT 

Aptitudes 

Directional 

Headings 

Dial 

Readini 

FSS 

18 

40 

(0) 

7 

VFR 

55 

10 

21 

(0) 

IFR 

49 

8 

13 

(0) 

ARTCC 

33 

34 

7 

5 

All  Options 

37 

21 

15 

(0) 

Note.  Data  are  from  Mies,  J.  M. , Colmen,  J.  G,  and  Domenech,  O. , 
May  1977,  Table  XV.  3 


All  data  were  collected  at  ATC  facilities.  Cor- 
relations between  selection  tests  anti  supervisory 
ratings  are  presented  in  Table  4.  Correlation 
between  MCAT  Conflict  and  Aptitude  scores  and 
supervisory  ratings  are  significant  (p.  <.01 ) for 
the  FSS  and  Terminal  VFR  options  and  for  all 
options  combined.  The  correlation  between 
MCAT  Conflict  scores  and  Terminal  IFR  super- 
visory ratings  also  was  significant  (p.  <.01 ). 
The  MCAT  Conflict  and  Aptitude  variables  were 
prominent  predictors  in  most  regression  equa- 
tions which  predicted  job  performance  “success,” 
although  neither  predicted  at  significant  levels  of 
confidence  for  the  Enroute  option.  Directional 
Headings  appeared  valid  only  for  the  VFR 
option  and  for  all  options  combined.  Dial  Read- 
ing entered  for  FSS  only. 

ATC  Progression.  ATC  Progression  was 
measured  by  comparison  of  the  ATC  option  to 
which  the  speciali.st  was  initially  assigned  when 
hired  to  the  option  assigned  on  .January  1,  1976. 
A progression  from  FSS  to  Terminal  VFR  and 
IFR  and  Enroute  options  was  used  to  reiire.sent 
levels  of  increasing  complexity.  A progression 
of  “high”  was  assigned  within  this  hierarchy 
when  a specialist  was  in  an  option  of  a com- 
plexity level  the  same  as  or  higher  than  the 
initial  option  assigned.  .V  progression  value  of 
“low”  was  assigned  when  a sfiecialist  was  in  an 
option  of  a lower  complexity  level  than  the 
option  to  which  initially  assigned.  Correlations 
lietween  selection  tests  and  progression  scores  are 


presented  in  Table  4.  Again  the  MCAT  Con- 
flicts and  .Vptitudes  were  prominent  predictors. 

Aggregate  Criterion.  An  aggregate  criterion 
of  .VTC  “success”  was  constructed  from  combina- 
tions of  the  four  individual  criteria  (training, 
on-the-job  performance,  progression,  and  attri- 
tion) and  provided  a five  point  scale  value  for 
-Vl'C  succe.s.s.  The  .Vrithinetic  Reasoning  Test 
was  valid  only  against  the  progression  criterion, 
and  was  dropped  from  the  battery.  The  Direc- 
tional Headings  Test  was  dropped  for  FSS.  since 
it  failed  to  enter  the  regression  for  supervisory 
assessment.  Correlations  between  selection  tests 
and  the  aggregate  criterion  are  pz-e.sented  in 
Table  4.  All  correlations  are  significant  (p. 
<.01 ) for  each  of  the  four  options  and  for  all 
options  combined.  The  Dial  Reading  Test  did 
not  enter  the  regression  for  the  VFR  option  and 
entered  last  Avith  a negative  “b”  weight  for  IFR 
and  for  all  options  combined.  The  integer 
weights  assigned  to  tests  are  summarized  in  Table 
5.  It  is  evident  that  ^ICAT  (Conflict  and  .Vpti- 
tude  segments)  is  the  major  factor  in  the  validi- 
ties (R)  derived  from  the  multiple  regression 
analysis. 

Summary 

.V  new  test  has  Izeen  developed  for  initial 
sc-reening  of  FAA  Air  Traffic  Controller  appli- 
cants. Its  content  includes  the  traditional  types 
of  aptitude  test  items  found  in  today’s  Civil 


Service  screening  battery.  In  tuldition.  it  in- 
cludes a measure  of  the  ability  to  identify 
potential  conflicts  in  air  traffic,  a skill  tlmt  lias 
been  demonstrated  exiierimentally  to  have  sijr- 
niflcant  relation  to  success  in  the  FAA  ATC 
speciality.  All  test  <[uestions  are  preesnted  in 
an  air  traffic  control  setting,  which  gives  them 
a job-related  appearance  not  found  in  today’s 
selection  battery.  Alternate  forms  of  the  test 
have  been  developed  in  a paiier-and-iiencil  for- 
mat to  meet  Civil  Service  needs  in  their  decen- 
tralized testing  program.  The  test  has  lieen 
administered  experimentally  to  groups  whose 
abilities  approximate  those  of  the  applicant  pop- 
ulation, and  correlations  between  alternate  forms 
indicate  that  it  has  satisfactory  reliability  char- 
acteristics. It  has  been  administered  experi- 
mentally to  students  entering  the  new  FAA  ATC 
Academy  and  personnel  on  the  job  at  operational 
facilities,  and  constantly  pro<lucpd  higher  corre- 


lations with  ATC  success  than  any  other  test  ! 

used  in  the  validation  .studies.  The  available  ! 

ilata  indicate  that  this  new  and  customized  in-  ' 

strument,  when  used  in  combination  with  other  i 

selected  measures,  promises  to  l)e  a significant  \ 

improvement  over  the  existing  battery  for  screen-  ' 

ing  FAA  Air  Traffic  Controller  applicants.  j 

I 
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